Quantization and the method of k -means

T HE THEORY developed in the statistical literature for the method of k-means can be applied to the study of optimal k-level vector quantizers. In this paper, I describe some of this theory, including a consistency theorem (Section II) and a central lim it theorem (Section IV) for k-means cluster centers. These results help to explain the behavior of optimal vector quantizers constructed from long stretches of ergodic training sequences. I also offer a new proof (Section III) for the consistency theorem, based on an identification of the optimal quantizer with the measure that m inimizes a Vasershtein-like distance between an emp irical measure and a collection of discrete measures corresponding to k-level quantizers. By a k-level quantizer I mean a map 4 from some Euclidean space IWd into a subset {a,, a2;. .,a,} of itself. Such a map can be used to convert a d-dimensional input signal X into an output q(X) that can take on at most k different values. An optimal k-level quantizer for a probability distribution P on lRd m inimizes the distortion, as measured by the mean-squared error P ] X q(X) 12, for a random vector X with distribution P. (Instead of the traditional symbol lE, I use P to denote expectations as well as probabilities. A similar convention applies for expectations with respect to the probability measure P.) O f course, this makes sense only if the expected squared Euclidean distance P ( x I2 is finite, or, equivalently, the L2 norm II XII = 2 PIXI ) ‘/* of the corresponding random vector X is finite. Such a constraint will remain on P throughout this paper. Searching for an optimal k-level quantizer for P is equivalent to the k-means problem: find a set A = { aI, a2,f ep ak} of cluster centers to m inimize the within cluster sum of squares